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(lexical, external consistency, and internal consistency)- the " 
remaining subjects were not given specific information as to the 
standards they should use. All problems subjects identified were ' 
classified as to the type of standard they reflected. The 
classification scheme -consisted of the three targeted standards plus 
syntax, prepositional cohesiveness, structural cohesi veness , and 
informational completeness. - Of particular concern were differences in 
the- standards adopted by ^students receiving specific instructions and 
those receiving general instructions. Use of the lexical standard did 
not differ with instructional specificity, suggesting that students 
spontaneously evaluate their; understanding of ^individual wards 
Students receiving, general instructions rarely used the external and 
internal consistency standards, suggesting these are' not criteria 
students typically adopt. Instead'; they commented frequently on the 
s|ruc|ural. cohesi veness of the passages. Students .with higher verbal 
atoility exhibited more frequent and mpre varied standard use than ' 
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'Abstract 



College stuiient|s differing ip veri% ability read and 'evaluated their 
^ comprehension of expository passages . Three -different types of problems, were 
embedded within 'the passages to provide opportunities for "students %o reveal 
the use of -different standards of evaluation. Half of the sufcjects were 

* • • • i ' - • : i • * • * - " • - : 

informed that they should use three particular standards in order/ to identify 
ther problems (lexical; external consistency,; and internal consistency) ; the 
remaining subjects were not given specific information as . to .the. standards 
they should use. All problems subjects identified t regardless. of whether o¥ 
. not they were • intentionally introduced, were classified as to the type of 
standard they reflected* .. The classification scheme consisted of the three 
targeted standards plus four others : syntax , proposi tional cohesiveness, 
structural, cohesi veness t and informational completeness. Of particular 
concern' were' differences in the standards adopted by students receiving 
specific instructions and those receiving general instructions* Use of the 
lexical standard did not differ with instructional specificity, suggesting 
that; students spontaneously evaluate their understanding of ihdividual words* 
Howeyer, students receiving general instructions rarely used the external and 
internal consistency standards, suggesting these are not criteria students - 
typically adopt. Instead, they commented frequently on the structural 
cohesiveness of the passages* Additionally, students" with higher verbal 
ability exhibited more* frequent and more varied standard use than those with 
lower verbal ability. V ■ 
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Imagine yc-u x^re a. coll eg 4 student reading a section ii>. one of your 
tex tbooks and ^.you ara. s jerious in your goal to, understand the material, Hqw 
woiCld jroy , dealer Whether or not you understand? that is, . what criteria' or! 
. standards would you use for evaluating your comprehension? There are probably 
■►. some criteria that virtually all students would use, while there may be others 
that are adopted by, .only a few*;! 1 For example, most students would probably 
decide they are having trouble understanding the material if they encounter 
many words whose meaning they do na^t know. Since the effect of vocabulary 
knowledge on comprehension is widely recognized. ; a lejtieal standard should be 
one which is frequently adopted. Suppose: the, passage contained information 
related tp something you had previously* learned* Would you consider how the 
new information relates to your pr iorf^nowl edge? "Or* would you accept the 
information at face value, ^rtever questioning the -truthfulness ^of the text? 
Unfortunately, the latter course may be more common! reflecting a failure to 
; evaluate text for external ^consistency* 

\ At the present time^. there is little empirical evidence upon which to 
* base an answer to the question posed above/, TJia't readers in fact evaluate 
their understanding is implicit in models of comprehension (e.g., Collins, 
Brown, 4 Lar kin, 1980; Just & Carpenter, 1980) and explicit In models of 
comprehension monitoring (e.g.* Baker Brown, in press ; Flavell, 1981; 
Lefebvre-Pinard & Pinard, in press; Mar kman, 1981). Nevertheless, theorists 
have been very vague as ■ to , how readers decide whether or not they understand* 
However, Baker (in pregs), has recently proposed tha t-evaluation involves the 
application of standards which all- into three basic classes: lexical , 

syntactic, \and- semantic. The semantic class is further subdivided into 

» ■ , - ■ ■ ■ ■ =- ' . . . ■ » 5 • . - ■ * ! . = • . " " * 
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s^nd^rds of external consistency; ,internal consistency , •prepositional 
ob^si|^Bfess f structural cohesi veness", and informational completeness: 
£^id€mg^5^;Qij&ed froro\ k wide variety of sources showing that readers are 
sensitive to manipulations that affect the eompr extensibility of text along 
each of these dimensions* 

The advantage of* considering evaluation with respect to d stinet 
dimensions becomes apparent when we examine the conclusions that have been 
drawn in research on comprehension monitoring, The typical, paradigm has been 
to introduce a particular type of problem into a passage and assess a reader's 
ability to detect it. Consider, for example, a passage containing the 

following sentences: "The temperature on Venus is much higher than bbiling 

."••*"• • . v .• i.* *• ■ '.*•..' ' ' '#.*••" ■■ . • ■ ' 

water* But it is much too cold fo^ us to live there," In order to realize 

. ' • ' •' % v ' .. - : ■■ •.. , . ' , • , - . 

that the ideas in these sentences are contradictory, the reader must evaluate 

the internal consistency of the text. Now consider the following: "States 

with many rotusts rely heavily on the sales tax," In order to realize that 

the meaning of a particular word is not known T the reader must apply a lexical 

standard of evaluation. Failures to notice both types of problem's have been 

attributed to ineffective comprehension monitoring (e.g. , Markman, 1979; Paris 

& Myers f - 1981), eveh though detection of the problems requires that two 

different standards of evaluation be applied - In other words, the resul ts 

reflect failures _to use particular standards of .evaluation ; they reveal 

nothing about the possible use of alternative standards. It may well be that 

the same person . who fails to use an internal, consistency standard would be 

very likely to f apply a lexical standard, - a ' 
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■ ^Several researchers have report, '«lo; )ra < = .,d reader proficiency 

differences, in comprehension aoriitor using t0 bal definition (e.g.', 

• • Garner,. 198O, 1981 r Mar kman.i 9791 H. " ■ '"'V / TerwogC i Visser- ^ ^8lV. 
Paris 4 Myers, 1981; Wihograd & , . -or. . 9 f. ,). .However .( comprehension 
monitoring should not be regarded as , — henomen^omething at which 

a reader is either "good" or "poor. u , are to help^tudents evaluate , 
their .understanding more ef f ectiveiy , /-we' need: Ato ' acquire >a better 
understanding of differences in the use of apeeifie standards, x The present , 
study was designed to provide a foundation for this effort.*: It addressed such 
questions as whether some standards are more likely to be applied than others 
arid whether more effective readers differ from less effective^ Readers in 
either the scope of their standard repertoire (i. el, the number of different 
standards used), or the composition of the repertoire (i.e., the specific 
standards used) , 

College students differing in verbal ability par tieipated^in a study 
whose stated purpose was to find out what kinds of things make passages hard 
to understand. The passages, excerpted from college-level textbooks, 
contained many potential sources of difficulty, covering the full range of 
stariaards described by Baker (in press). In addition, three types of problems 
were intentionally, embedded within the passages, problems which could be . 
detected. by using the lexical, external, consistency , and internal consistency 
standards, of evaluation.' Half of the; subjects within each verbal ability , , 
group were specifically told that these three types of problems would ; be : 
present, and they were given, examples of each type. The" remaining subjects 
were simply told that' problems would be -present. Of particular interest here ' 
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•: W3S Whethcr ^6 subjeets receiving 'general instructions would be less - likely 
to, a/ly any of" the targeted standard, than those receiving specific! 
• ^^uotiohs and whether - they would be more likely to spontaneously adopt any; 
, ; of the *en-targeted standards. Also of concern' were possible interactions :■ 
.'; with Verbal ability level. 

The primary focu^ of the paper will be on the usd of d iffer eh t; standards 
\ rather than on the ide'nMf ieation of embedded; problems. ] ' This focus reflects a 
:, .shift from Previous Vesea^ 

success at detecting specific problems. There .are several reasons, for the 
- shift. The first, .mentioned earlier, is ttfat failure "to notice a specif ib ' 
: type of problem reflects a failure to use a specify standard of evaluation. 
' n P t a failure to monitor comprehension,* A second reason is that failures, to 
notice particular %roMems« may j>i; due to a number of different factors/ 
• including, lack Jof relevant prior knowledge, misinterpj- etations, and 
inferential^asoning that serves to resolve the praWao^cT; Baker , 1979): • 
It seemed mortyensible, therefore, to focus on wythint a particular subject 
perceived as problematic, regardless of whether ;or , not / the experimenter - 
perceived it ^ueh:, This approach affords a much richer perspective on the 
ways that readers evaluate their understanding. A third reason for the shift" 
Is that ■differences in; detection of the different problem types are difficult 
, to interpret (cf. i^kman« > Gorln, 1981)1 Since the problem types are 
quaiitativeay different; from one. another, we do not know whether toey would be 
equally ensy to .detect if the appropriate standards were used effectively. 

Xx^ 
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Subjects v ■ f " ; 1 ' 



v 



men. 



A. total >f 5?. college students participated in the stud^. 37 women end 22 
The •studepts were enrolled in n„^ introductory, psychology course end 
Participated in brder, ^^^ciidtt, The majority, of the sheets .' 
■ were first-semester freshmen/ In order to obtain two groups of subjects who" 
■ differed cprislde^bly In % v^ ^^ lBV el, onl* |those students whose:' 
SAT-Verbal '\-^^«^^L^-"«Mfc^^"_^(i _ «M^^^~^^- ~ to participate. The 
final sample consisted of 23>tuden^| h |higher verbal ability (mean-599, • 
s.d.i66) and. 35 wfth 'lower (n,ean s 388. ^2>V ; The inequality of group 
sizes was not apparent until some time afterpe data were collected; several 
students reported that ^eir scores were in the high category, but in fact 
their scores were low. All subjects were asked at the outset of the 
experiment to sign a ! form giving permission, for their SAT scores to be • 
released from the admissions, office. When; Ah e tf fieiar scores were received, 
'several^tudents: ^ to be reclassif i ed . Additional . data collected on each - 
subject; were the number of semesters enrblled in college; VKo^lrtiv.il^iv 
introductory psychology ooursVgrade, and Math , SAT score, ^ ^ ; : , " 



, hn J hf \ sl sn-up„ sheet did not specify that only native speakers* of English 
Verbal Iat\ UP " COr "^ Uently ' one "on-native speaker did part ci^te 
Verbal score was 230, but his performance on the detection task was kettir 
than any other subject. Moreover, his Math SAT score wis »Sl 
introductory psychology grade- was an A. Since his VerbaT score was "clear! I 
analysed ■ Sentatlye ; 0f h H ^litie T , ^his subject was dripped from "the 
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Subjects were assigned to either the general or specific instruction** 
condition when, they reported to the laboratory. An attempt "was made to keep 
the conditioners balanced as possible* for verbal ability level and gender 
throughout the course of testing. Jn order- to prevent possible bias in 
testing f the subjects were asked to I wait until the ejpd* of the experiment ^tq, x 
report their* SAT scores. At the beginning of the session they "were sitaply 
asked whether "their score did meet the requirements" specified on the subject 
sign-up sheet. 

• : • \'. * . .- . • . - .. . . . - ;. 
Materials . 9 ; K • ■ 

1 The ^materials consisted of 12 expository passages adapted from 

college-level textbooks, , Four selections were taken from each of three 

different texts, one dealing with animal behavior t a second with state and 

local government f and a third wi th environmental/ecologieal^issues . Examples 

of passage topics from each book, respectively, are: temperature regulation 

in lizards, advantages of the sales tax, and* the disposal of solid wastes- 

The topics" were selected to ^contain facts hot generally known and not likely 

: to >e encountered in - the beginning college student's ^coursework. . The 

selections were shortened, if necessary, so that all of the passages would be 

roughly comparable in length, Each passage had 11 or 12 sentences; the mean 

number of words was 202 (s.d*. =16, 48) , The readability of the passages was 

measured , using Fry's (1978) scale; however , no attempt was made to modify any 

passage's readability level* (mean grade lev el =11. 67 • s .d , =2.50) 

< The passages varied- considerably in style and structure, as well as in 

/the quality of writing, thus affording ample opportunities for the students to 

' question* various aspects of the passages. However, it was felt inappropriate 
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to rely exclusively on naturally-occurring opportuni ties to identify 
problematic sections of the passages therefore nine of the passages were 
intentionally disrupted S The disruptions consisted, of nonsense words, prior 
.knowledge violations, and inconsistencies. These different problem types 
could be identified by effective use of the lexical, external .consistency and 
internal consistency standards of evaluation, respectively. One of each 
problem' type was • present in each of the three content areas. All of the 
problems were created by replacing a single word with another word, The 
substitutions occur red in the middle of a sentence which appeared in any 
serial, position from . five through eight. Three of the modified passages 
contained two-syllable nonsense words which followed standard rules of English 
orthography. An example of a nonsense word appears in the following sentence: 
"The landfills must not be in eolactins (locations) that could lead through 
seepage to contamination of ground waters" Three other passages presented 
facets that violated common .world knowledge. An example is: "During the 
summer (winter) months most frogs hibernate in the soft mud of the bottom of 
pools and streams," The third "type of problem, also, embedded in three 
passages, was internal lneo'nsisten#, created by making one sentence in the 
passage conflict with a previously presented sentence. One: sentenca'TSays 
intervened between these two target sentences. An example follows: "These 
small [housing] firms usually assemble , the ' principal components of the house 
themselves. The reasons for the differences between, these two industries 

• - ^^^^^^^ : * ' 

Chousing and - autbl lie in the fact that local government is the primary 
regulator of land use-in this country. Zoning and building code regulations 
are responsible for the presence (absence)' of large mass-production oriented 
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construction firms." The remaining three passages * did not eonfifein any 
intentionally-introduced problems. A fifth selection from one of the texts 
served as a warm-up passage- ^ " ? V v 

A preliminary screening study was carri-ed out in which 23 advanced 
undergraduates Were instructed to r ead- the passages carefully v and to' comment 
on atiy aspects that seemed problematic . They were* specifically told that 
problems, were present and were given examples of the three types that, had been 
introduced, 'but they were given, no hint as to how many problems might be 
present in a given passage. The purpose of the study was threefold* 1) to 
ensure that there were neither ceiling nor floor effects in the deteetability 
of any problems; 2) to ensure that there was not extensive variability in the 
deteetability of the different exemplars of a given problem type; and 3) to 
verify that there would be other sections of the passages that individual 
"subjects would identify as problematic. Subjects 1 responses were scored both 
for problem identification and' for the other kinds of comments they made* 
Detection rates ranged from* 20%* to t 80% f neither at ceiling nor floor* 
Inconsistencies and prior knowledge violations averaged 30% f nonsense words 
70%* Within problem types, there was some but not extensive variability. 
Subjects did indeed identify many additional parts of the passages as 
probl ematic* For the most part, these additional comments. were idiosyncratic, 
though occasionally two or three people had the same comment, Moreover the 
comments reflected a full range of standard use ; subjects did not restrict 
themselves to the three standards that were needed to detect the intended 
problems* Also present were comments about syntax, structural and 
propositional cohesiveness, and' inf^rraatiortal clarity and completeness. 
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The -final versions of the passages were ass ambled Into booklets. .Each 
passage -was typed on a' separate sheet of paper 'and was .heeded by Aeserlptive, 
title. All passages appeared as ; aj sUngle lo^g paragraph t %^y^'l£<jippe 
paragraph^ had been present in the [wi^iiii_ : %^-.\'\:^iM^ihmi\.''J^B r , gives 
> subjects occasion to comment* on 'the structural' oohesiveness of the text.) At 
the bottom of each" page was a set of four evaluative sentences which were to 
be used by .the ' subjects .'to indicate whether^ the passage was -problematic-, and X 
• if so, to what extent did it affect-' eoniprehensibility (see Procedure section) 
The same passage ; appeared , in; first position in each booklet and served as a 
practice passage. The remaining 12 passages yere arranged in a constrained, 
partially randomized, order. The eons fcraint was that a particular problem 
type had to- appear in a particular position across booklets , but the passage 

- ' • > • . • •' 

■_'%■' J.- ' ~ \ ' . . • - . " . " . • " "'-.=* 

containing that problem was randomly selected from the pool* " For example,; the 
first experimental passage contained a prior knowledge violation; for snyt 
given subject £t could be any. one of the three prior knowledge passages. The 

second passage always contained ah internal inconsisteney , and it tob Wulii^fre • 

; \ ■ . . ■ » ,. ; ' J . '■' , .; . \ • . . . .JjT' : ■ ■ 

any one of three, etc, ' - * " . . v " v -\=. 

•j • - t ' 1 - ■- t 

Procedure ' '=■ • "v-\\: - ■ '. " . . '•■ ■ 

All subjects were seen individually, by the author V At the ■ beginning of - 
each session, the subjeets^were asked if they would /tie willing to authorise 
release of the^r* official test scores and if so, to sign a consent form, (All 1 
subjects were willing,;) They were then given a > booklet* containing the 
passages and were asked to fill in on the, cover, she^t their names, ages, dates 
of birth, and number of semesters they had attend the bottom of*' 

the page were four statements that the subject;? were asked to refer to while 
they listened to the relevant portions of the instruotions 



> The instructions were presented on tipt, recorded by, the experimenter. 
Subjects in bo tli conditions wef © t»ld >tbtat the:: purpose ^ 
:^..':^ n t <?f toingl mi^e passages hard to unders^riii ; were 

^hsfruete^te^ wa3 to identify problems that had been 'deliberately, 

introduced In to^ some exposi tory passages static mighty find iitt" 

college tex tbooks . ; • ,.A probl em ' was Bif ined as "something that might confuse 
" people or something that people might have trouble understanding*" Subjects 
iS^^S: sp ® c 5 f ^p further inf or ^ 

-exagt nature the problems and two examples of each type . The examples of 
the internal i neon si s tenci e^a rid prior "knowl edge viplations were the same as 
those uaed by Jiarlqnan and Gbrin; (1981) ; the nonsense word examples followed 
the same format. The terms used to descr 

two parts of the paragraph that don 1 1 make sense together 2) things that 
aren f t true;- an4 3)* words that aren't really words. Subjects were not told 
^pw many 'problems might be present in any one passage. ; ^ 
s / Subjects in both conditions were instructed to underline anything they 
thought was a problem as they were reading. Then, when they finished/ they 
were to check one of the statements at .the bottom of the page in the space 
provided. The statements were: 1) There were no problems ; 2) There may have 
been a problem but it did not make the passage har<d ' to understand;;- 3) There. 
^ as a problem and it 9 made the passage a little hard to understand; and 4) / 
There was a problen arid it made, the passage harder to understand. (As it 
turned but; there were large individual variations in the : us^ of the rating 
scale; some subjects only used the first two statements, others used the full 
range. For this reason, the rating data were not subjected to any statistical 



analyses. ) The subjects were f ur ther informed' that after they checked 
"-" statement they wer e ^ 

underlined any wards ©r phrases. / V' ^ ' - - 

=*^f*t *lie .ihs^u^tiona were presented f the subjects were asked W turn to, 
-\ th'e..-. firs it pr ae tic e ) passage ; and read it silently to* themselves." Care- Was 

taken to erisure that the Subjects understood the^ task by reminding them* if 

necessary, to underline problematic information as soon as they encountered it 

a ?d by pointing out 

(••$-• • statement #3 Was checked j>ut no sentences were underlined). The 
subjects were not ; given any feedback that they undef stood 

the task, 'ibhe. subjects went oft. to the 12 experimental passages. For each 
passage , the sub j ects read ; i t siren tly at their own pac e t und er 1 ining any 
problematic sections as they went along. When they finished reading the 

■■■■■"<■ - • •' - m •>• • '. • ' • ■ •• •••• - • * ' •-. • " 

• ' ■ ' ■ - • . . . - ffm • • . • " • • • • . ' . ;• r -. * . . • - < 

passage ,. they checked one of the rating sentences , and'* then explained why they 
had done so and, if appropriate, why they underlined what they did.* r 

. Upon completion of the, task, the subjects were asked a standard set of 
questions designed to reveal how they had interpreted any problems they did, 
not report. Before asking each question, . the experimenter turned to the 
appropriate passage and placed it in front of the subject, who was encouraged 
to reread it to find the answer . If the subject did not spontaneously 
indicate awareness of the problem after answering the question, the 
•experimenter asked, " Bo es that make sense to you?" If the subject still -did 
not indicate problem awareness, a second prepared question was asked. To 
illustrate, the questions for the prior knowledge problem embedded in the 
sen tenee, ; "During the sunimer months,, most frogs hibernate in* th^sof t mUd.,^- 
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;We f? as f olio w^r When do frogs hibernate? an^ Do frogs -typically hibernate 
^ ' ^; Ail. sessions were tape record ed> arid the tapes were subsequently 



« ... 

transpribe^l '.^evVl^rigth''.^ 



per sul^H^ wi th a range of 2^Vtb 60 minutes , 



Scoring ^ ' ; Y:\- : *':•// : *;- "."'V- ,". ''. / ' 

. ; Prbbl em i d e n ti f ioa tion , Responses wer e snored as problem identifications 

explanation of the nature of the problem/; I 

subject to speedy the! problem type. So/ for example, if a subject underlined 
the nonsense word colactins and then said , " I ' jdon f t know wha t thatfc word 
means v« the response toss 

subject explained that she underlined the phrase, "during the summer months," 
because she didn't think frogs hibernated in summer, this was scored as 
correct. There was never any question on. the basis of the responses as to 
whether the subject noticed the intended problems * 

trariscrihed, the subjects 1 responses to "non-problematic" segments of text 
were coded as to the type of standard they revealed. The coding scheme used 
was that established by Baker (in press) and -included, .seven categories: 
Jextpai » f external consistency, internal consistency, syntax ,Mnf or mational 
Giarity and completeness, prepositional cohesi veness f and st^Jctural 
ephesiveness. Examples of comments reflecting the use of these standar\s are 
presented in jabi e l f along with the relevant passage^ con texts; 
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It .is impo^ant. to note> that no judgments sere made as to whether 
C in^rrtatiori w^g In faet probleoatio. For example/ if a subject misuhderaWod 
a particular sentence and so said that the following sentence was 
inednsisteritf this was scored as an application* of the internal consistency 
, *£an^ 

protocols were scored by two independent judges who were able to classify all 
but 2% of the comments ^ into one v of the seven categories V In ter- judge 
agreement was^ high and the few discrepancies were resolved through discussion. 

.. ^ ... .. • •. • - - - • . ; ' : ■ . '. "• '"" Results '. ' "' ' .. "'• ' • 

This section of the paper is divided into several different sections^ 
The first section presents the results for the problem identification task, 
•reporting both an analysis of variance "and a multiple regression analysis. 
The second, most important, section focuses on the application of the various 
standards as the subjects- were reading the entire passages, Several different 
dependent measures: will be examined: 1) frequency of use, 2) number of 
different standards used f and 3) presence or absence of par ticular standards 
in a subject's repertoire. The third section will briefly report the more 
fall en t >3peets of subjects responses to_ the follow-up questions v about 
probi ems they failed to report. •:' ._ , - -A/ ■ s 

Problem Identification : /• 

The mean number of problems of each type that the students identified is 
shown in. Table 2. Quite clearly, the students did hot do very well on j this 



task. An analysis r o^; Variance with instruction condition and verbal ability 
l^v^i v^t^^n-^ubjaot factors ana* problem type a£ a • wi thiniaubjeot factor 
^fY?al^ fe^iablf raain*effeots of each* factor as well as an: instruction by 

-?y;pf^ to 

' ° f • i identified morT problems 
those who were simply informed that problems would be present (25%) , F (1, 5*1) 
= 20.85, jp ^ .001, V Additionally, : students who scored higher; on the Verbal 

(27%) • F (1, 54) = 13-02, £ < .001. Interpretation of .these main effeets is 
qualified by the in higher verbal 

ability students who received general instructions did not identify more 
problems than the lower ver bal "ability students receiving general 
instructions . Moreover , the performance of this higher /ability group did not 



' Insert Table 2 about here - ■■■■■ 

differ from thatt of the lower ability students who received specific 
instructions. ^/Although specific instructions were associated with better 
problem " identification - for both higher and lower ability students , the 
magnitude of the; difference between instruction, types was much greater for the 
higher ability group* • r -/ 

The identification rates for the different' pr obi eta types differed , as L 
expected, F (2, 108) - 76-79, £ < .001. Many more nonsense words were 
identified than either prior knowledge ^violations or inconsistencies C66%* vs. 
21* vs. 10$, respectively). The diff ere^o^ between the latter two pr obi Ira 



types was also statistically significant - (Fisher's -lad-. s .28) v There- were no 
li^eratf^diia of < l ) ,K 

• 5 V The ab^ the data was 'based on a dichotoraous elpssif ic^tiori: 

„ of , iHibieeits; into higher and lower verbal ability levels . ;The r eliable main^ 

" effect >f verbal ability indicates that the two groups did indeed "differ* But 
it does not indicate how "much of the variance in problem * iden tif ica tion - is 
attributable to verb^L" ability * T6 answer : this question , multiple, r egr ession 
analyses,: were^^^ 

predictor 'variable, : Additional predictor variables were Hath SAT score. 
In trodue tor y*; Psychology final grade , age, and number ofV semesters enrolled in 
college. Because instruction condition Was a qualitative variable, the data 
were an al y z ed $ epar a tel y for the two conditions. The. total number of problems 
identified; served as ; the dependent variable. ^ - 

f ■ . v . - . . j . ■■ . .. •; . . ' ; \ - ■ > :. • . • ,. . . ■ -" : , • * 

' The regression analyses for the two different instruction conditions 

yielded very different ^outedraes^ as might be expected given the analysis of 

variance results* We will consider first subjects In the specific instruction^ 



. ^ ■ • 



condition. The univariate' correlations with problem identification for the 
Verbal SAT, Math SAT/ and course grade, respectively, were •72, .67, and *39, 
However, when the effects of the Verbal s^re were partiailed out, neither the 
Math score nbi^ the, course graie correlated reliably with problem iden tif ica^r 
tipn. / The predictor variables were entered into the regressipn equation 

through a forward stepping algorithm which required a minimum F-ratio of 4,0 

♦ * -• 'i V .""*■'.•. .. - . , . ' • . . 

for entiry. Verbal score was the only variable to enter the regression 

equation (F to enter m > 25, 39) , accounting, for a substantial proportion of the 
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%- ability had virtually no predictive value forj 

Univariate correlations /for Verbal : SAT f - Mattj^ 



pro b 1 em ; i diten t ijf ie a fci oh /« 



SAT v. and ; eour sev^gr Sdi 



respectively, were *l** f *36 % and 4O3. None: of the variables had sufficient 
F— levels to* enter the regression equation » though ; th e Math SAT spore was* 
close, with an F to enter of. 3-91- Even ,thisWvariable f however , accounted for 
but; 10% o^^ 

Thus , 1 the regression analyses lend fur ther ^suppor t. to # the c^i^lusiqn 
drawn on th^ 

has strong effect on students 1 ability to- identify problems wh eri^ they are • 




specifically instructed what to look" for v it has -virtually ^ no . effect on 
problem identification^ when the problem type3 are not specif ie^Vy 
Application of Standards Throughout th e Tes ti hg Ses sio h / 1 ■'■ 

As indicated in the Method sec tion t the; data to t>e considered in the' 
.following analyses are based on all problems that the students identified." 
regardless of whether they had, been intentionally introduced by the . 
experimenter * .Moreover "; these date are based on all 12 passages Che subjects 



read, not just the 9 that contained intentional -problems* Several different 
questions about standard use Will be addressed. The first concerns the 
frequency with which subjects used each standard, the second concerns, the 
number of different standards each subject used, and the third concerns 
patterns of standard use, i.e., whether there were some standards that 
subjects never used at all or used to the exclusion of any others. 

Frequency of use . The mean number of times students applied each* 
standard is pr%sent^d in Table 3 as a function of instruction type and verbal 
ability level. "Note "that- the- standards differed considerably in their 
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frequency r of use , as Expected. Nott also that the pat terV differ^ depending 



too whlether 

(instrUQtioti) x 2 (Verbal ability) x 7 (type of standard) mixed analysis/of 
virianoe confirmed, Tthese apparent differences. ; The main ef f eeVro^f -at^ndar'd: 
type V wa^ r el iabl e f > ^ (6 , 324) = 21.66,- ^ ; iO^f^s was the standard by 

\' ; : •,. .'. : . •. • , •■.-■-=-.-•- :...-*-•-..* ' ••• ; v • v-V,--.-. ^^r; < ; ..." i ■, ■■ ; 1 .- . . .= - • 1 . .. ' .. '. •,- 

V •- •• ••: - - • : ' . • * ' S> v.. - - - . 

instruction interaction t £ C6, 324) = 9.11 , £ ;< .001. In addition, the lower 
verbal abili ty groups overall used the standa^s less frequently, F (1, - 
il ji| i| . ^ 05 . Ther e wa s al so a tend ency to war d s a thr ee-way in ter ac tioff , T 

(6, 324) .-:= 1.81* £ < viO* V . .V ; -?- : "'"- ; - " - 



Insert Table 3 about here ; ' m 



r One might have predicted that subjects receiving specific Instructions 
would 'apply "the three instructed standards more ^frequently than subjects 
receiving general instructions, leading to a main effect of instruction 
condition. However, the main effect was not reliable, F (1, 54) = 1.27, JP > 
.10, although there was the interaction -With .standard type. Fisher f s Isd 
procedure wis used to test the differences between, means (isd = .68, £ < .05). 
Specific instructions did ; in fact lead to increased use of the external and 
internal consistency standards, but the lexical standard was used equally 
often, by both groups of subjects. One of the noninstructed ^standards, 
structural cohesiveness, was used more often by subjects receiving general 
instructions. The r emaining^t^nee standards were used with equal frequency by 



" Of particular interest are \thm^ types of standards the students adopted 

most frequently when simply given general instructions to find the prGbiemsV 

■ v r-; 7 ■ ■ ' > V • •■ v '• •••• . - O, '9. .r ^- ; ;v, . ; , • %-= 

Si ?®^\i h ? s Imj^f^tions about how students would s^rita^eou sly!; ^yilutft^ 

^heir under Stan to process tex^t /carefully i The ; iexie^i- 

standard and* the structural cohesiveness s tandard were used more c^tek - than 

aT W others, ' Note that the mean for the lexical standard is greater than 3* 

• . / "•• - < < N * • - j . / j* *" - - * _ • ■ * /. - •■ - _• . • •• • 

which would be ei^e^ed^xf students reported all of the nonsense' words and 
receiving general i©s truotions identified fewer nonsense /worcis ^th^n Subjects 



receiving specific instructions ; yet they used t^e lexical {sta^ard equally 
of ten \. It: ^ € ® m3 v l*ft a J? • th ?y were particularly sensitive ^ 

items whbse meaning ;they were- not sure of while students receiving specific 

: •' , ' : I . • • • \ ' ' < ' . • ' j'. ; . . ' ■ •.' • ' , - . t-V;_-" " . . ' ■ - . . 

. instructions Tet passwords they recognised but- perhaps could 'hp t^ define. ^ 

* ''• . . • : •. . •. " , - - ; ' ' : '• ' i ■ ' - ' . ■ •' • 

TheJ high .frequency " of use of the structural' cohesiveness standard 

indicates that not only is^ comprehension affected by text structure 

^ har a c teni s ti cs # as many, have argued (e.g. f Ki eras, 1980; Kintsch & Yarbrough, r 

1982) v btit also that students are conscious of^its effects. It is interesting 

to note that the lower verb ability students used the standard considerably 

less often than the higheh- : This is in line wi th evidence that poorer readers 

' ' . • • • • " :;> . • ' . . ' . . ..• - • >. * '. ' •* '' ■' 

tend to be less sensitive/ to |text structure disruptions than better readers 
. (e.g. , Eamon t 1978/79; Meyer f Brandt, 4 Bluth, 1980). ,\ 'V: / ; 

Another standard that the general instruction subjects usted relatively 

of teji was the informational clarity and- completeness standard.. It was used 

• • ■• = • ■■ ' ' ■v;'-.;-' -, . , . ' : l .... ■' 

leis often than the lexical or structural standards but more Ibfte'h than the 

others, Here again, the higher ability students/ tended to ufs the "standard 
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more often than the lower ability students, suggesting greater concern mwith 
^whether Vthe Information ;was sufficient ^to enaW the 

autbor 'Was trying to make. ■ . \ ' : ^=/% v r;'V'-\.~ ::- >v i'~ ' "* \ M '. v'fjv " v '";.v ' V- 
*.\ . None pf : trie • o ther standards were used more than once during the entire 

session i ; by tfie average subject. / It is particularly noteworthy that ; the 
'ex ternal consistency standard falls ? into this group. " In a similar type of 

s tudy a lnvo3rgd ng f oqr th and Sixth graders, Baker (Note 1) found that the 



external consistency standard was spontaneously adopted almost as of ten as the 
lexical standard* Other studies have also -found that children are quite apt 
to challenge the truth of passage information (e.g.* MartonanV 1979) . The 
present f indihgs ^suggest that as students mature they become ..mor e l,ikeiy to 
accept information as given, wi thput critical evaluation of its truthfulness* 

1 . ■ « 1 . " •*» : . • *".«.= - ■ 1 * 7*i • .* 

■ . - ,7 : -7 . ; ; ; 7v •. ■ . • • . t ■_ , L ,. .... * . . . • • * _ ■ -■ ■ ■ 

However, that students will indeed apply the standard when instructed to do so 
is apparent by the large difference between subjects in the general and 
specific conditions. :...». 
I-/: We turn rtow to more detailed consideration of differences ^n standard use 
among the instructed subjects. The lexical and external consistency standards 
were used most frequently and equally often.- However, the lower verbal 
ability students seem less likely to challenge passage truthfulness than the 
higher verbal students. Surprisingly, they also * identified fewer word level 
problems. This may have indicated a reluctance to identify words as nonsense 
for fear that they might in fact be real words they should have known. The 
ihternal consistency standard was used less often than the other two, and here 
again, the lower verbal group used the* standard less often than the higher 
verbal group. Somewhat unexpectedly, two oth err standards were used as 
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* frequently as the internal consistency standard: „ structural cohesiveness and 
■ inforoational clari ty/co'mplet.eness . In otter words',^ the subjects did not 
: . . 1 imi t . their - commen ts to problems d esor itetf^Ah Jhe/ ^ns tr^c^ions^ • The 

, informational standard was used somewhat more' often by the' higher verbal 

^den^ for toe ;^itural 

., Standard; : "\- : r--' : V /";;V\ : =>':\ \*-.-.y.~J. v.--. V. . .. *7-V-. . : ... V - " •"' -v ." V -.3' : v=. . '•' 

• Wumber, of different ^h Aara^W^ ^^fe/in^eiriiii^. here was ■whether 1 or not 
" ; studerits; in " the dBf^'^^ 

evaluated their understanding. The number of different -s taridards ^u^rt /.'a l ■■ 
,. least once by^eaeh student was .. .entered into a 2 (instruction) x 2 (verbal 
^^^iMfe of :ywAB^a^JM^m^^ya^r-«y^ ed a reliable main eff ect 
of verbal abilil* leye^^ , The Higher verbal 

^ students, used - more; different standards than did toe low verbal; students 
(means ■= iflai : and 3.53. respectively) , This indicates ^hafr not only do the 
less effective readera report ; f^ as shown in 

. « the ^ 

against wh lc h to evaluate their understanding. The main ^effect of instruction ''" 



type was not reliable ( £ = .1^) nor Was the interaction withHerbai; ability (F 
< I)-, Thus, even those subjects who were not given any hint that there were 
three, kinds of problems they should look for spontaneously^applied an average 
of 3.61 different standards (as compared to H.13 for the informed subjects). 
Patterns of standard use. Another examination of group differences in 
■ the kinds of standards used was carried out by first classifying the subjects" 
as to whether or not they ever used a par ttieular standard and then entering 
the cell frequencies into mul tiWay tables. Tests of association for patterns . 
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of standard use were oarried out using a log linear model. The relevant data 
ar0 Presented in Table M as propor tions ; of subjects in each cell who used each : 
standard; (rather .'; than as number of subjects ; who did >^ 
standard)* ' ' • >- ' :: ; ; '='"/.:'- v .' . .;':'.\ ; ;"';"; : ; ; : ' = : '\V- : : . ".' 



3ta 
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^ We will consider V/^*\r«Hf^S^-vinb^ ;tfa^~^i^^aBrid|i ' i^/^sartf^ with respect to 
the lexical standard., /there were very few sub^eetsV in" any Condition; who never 
used it; tests of association revealed no differences. .The external 
consistency standard was adopted. by most of the subjects in the specific 
instruetion^^ups, but there were a majority in the general groups, who did 
not use it.^^l) a^20iil0, p_ < .001. The internal /consistency standard Was 
also adopted by more subjects in the specific group than the general group, X 2 
(1) = 7.78, p_ < .01. In addition, more higher" verbal subjects used th%* 
LstandarcLJJia^ lower l verbal, -]^-m =^ 
interact. " : ••' V , • '" ;' " •"' ■''*.''. 

Fewer than half the subjects in any^gsroup adopted the propositional 
cohesivehess standard. None of the tests of association were reliable. The 
structural eohesiveness standard, in contrast, was adopted by lost of the 
subjects in each group; again, the ^association tests ware nonsignificant. 
Somewhat more, than half the subjects adopted the informational completeness 
standard, with no differences -in group frequency. Finally, with regard to the 
syntactic standard," virtually none of the subjects receiving specific 
instructions used this standard, while a^ quarter of those; receiving general 
instructions did, X 2 (i) m 7.03, £. < ,01, ; , . 
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vr " : t^"^^" " : ^ T v '"'" > '/ J ■ v' • ^y-' *;cy; : ^v . - ; •/ . • • '^^-""^--^ : ^^ : -T^' ;> ""C 

Trf summary,, there were large differences aiao^g the standards in ^ the 
'likelihood; that subjects would adopt them, f The differ ences^ between ' 
insti'UQtioh Gondi tiding In the use of the ex ternal arid in ter nal ^orisi^ieriay^ -v 
b tandards ; ar e no t surpri sing given the experimen tal manipulation. Hpweyerv^^^^^- 
,what is surprising is that the dif f er ences wer e so large. This 3ugg es ti; 4 triaV f ^ 
a substantial proportion of adult readers -never evaluate their understanding •'• 
with respect tOY either of thesi two important criteria */ Although subjects : r 



with general instructions were pore likery to a^pt the syntactic standard 
than subjects "with specific insj^ructions, the proportion was low relative to 
^the other standards « indicating that these subjects too were typically more 
- concerned with higher-level sem*an tic features of ; the text. ; ^ln- particular f 
there were two . semantic standards that most of these subjects adopted 1 
structural cqhesiveness and informational clarity /completeness; 

The lexical standard, which was used by virtually all subjects, is not 
necessarily adaptive if it is used at the expense- of 'higher -level standards. 
, There is some evidence that ypunger and poorer readers in elementary school 
tend to rely exclusively on this standard (e,g #9 Baker, Note 1; Garner, 1981). 
It was of interest to ascertain whether less successful students 
college level also over-rely, on this standard. We have already seen that 
students with lower verbal ability use fewer different standards than students 
with higher verbal ability. What standards did they in fact use? This 
question was answered by focusing on those subjects who used only one or two 
different standard-is* Single standard use was very rare, observed in only y 
three subjects. Two of these subjects were in the lower ability group (one 
received general instructions t one specific) and the single standard lised was 
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the lexical* " (The third ; sub jeot, in the higher ability-general group, used 
the^ structural eoheaiveness standard exclusively*) • Therm were seven subject^ 
who used two standards ohly^ for six of these subjects* the lexical standard 
Waau dn# 7"bj?-<; the !$w© u$€d * ; Four y ef these subjects wer e in the lower— general 
group, \ and two in the lower -specif ic - (The seventh subject f in the 
high-general grbupy us^d the structural cohesiveness and informational 
completeness standards'. ) ^ Overall f 17% of the subjects had a repertoire 
limited to bne or two standards. Though this" is hardly a majority of the 
subjects, it is- significant that 80% of this group had lower verbal ability: 
and the same 80% always included the lexical standard in. their repertoire, 
ThiisV even among more ma tur e "readers ; those. ; who are less proficient show some 
overrelianee on the lexical' standard. 

Responses to follow-up questions . ! " * • . " • 

The follow-up questions were. included to reveal how students interpreted 
the intentionally-introduced problems they failed to report. By and large, 
interpretations jtfere those intended by the experimenter, indicating that 
the students failed to report the, problems because they did not use the 
appropriate standard when it was needed . ; The typical comment, upon being 
confronted with a problem, was, "I can^ t believe I missed that!" There were a- 
few exceptions, however, indicating not a failure to use a standard but rather 
a failure to perceive a problem. This was most, apparent with one. of the prior 
knowledge problems, embedded within the following sentence: "As governor of' 
Montana, Ronald Reagan would typically invite legislators to his house. , ." 
lAraong the: priof* knowledge problems, this was in fact the one most frequently 
detected (£ = '. .38)* However, subsequent questioning of the non-detecting 
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subjects revealed that 24 % of them .were unaware that Reagan Had not been 
governor of Montana. Moreover, all: but one of these nine subjects were in the 
lower verbal ability group. Thus, no t oft! y air e s tud eft is of lower verbal 
ability less likely to evaluate text for consistency with prior knowledge, 
they also have a more impoverished knowledge >ase- Unf or tuhateiy^ this latter 
^problem is less amenable to instructional intervention than the farmer (^^H 
press) . Failur es to perceive the internal inconsis as 
problematic were also more 'character istiei of the rower verbal group than the 
higher - Here the dif f icul ty is in logical reasoning, which again requires a 
differ 0 

effective evaluation of one's understanding. In short, the responses to the 
follow-up questions make " it clear that the: poorer problem identification by 
; the lower ability studenjts is not simply a produe t ; of 1 ess frequent or less 
effective use of the targeted standards of evaluation. Mor.e fundamental 
deficits in background knowledge and logical reasoning also play a role. 
'. **•• -v :.. •; Discussion J 

Jn this section we will return to the question posed in the openirig 
paragraph of the intr^cutioh a^ut how: would evaluate his or her 

understanding. The resul.ts of the present study offer many pieces of a; 
comprehensive answer. As expected, it was found that readers- do in fact use v 
several different standards during a single interaction with a text. Even 1 
when given no hints as to what standards should be used, the average student 
employed 3.8 different standards. This indicates quite clearly that studies 
which examine the use of only a single standard may -seriously under Estimate 
the flexibility; of students' evaluation /activities. Nevertheless , students of 
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.lower verbal ability did have more limited repertoires than their higher 
verbal ability peers. These limited repertoires tended to include the lexical 
standards a more primitive standard that can be applied without regard to 
'. * surrounding -context. Although this Standard should be included in any . 

reader f s repertoire (and was in fact used by 95? of the subjeets), itvshouid = 
T - b « which taki :: 

y : ^-- J i^% : . account -larger /segments of the tex t . : - '■■ rj;.>:-,j:LS^r^ 

; r ^ frequency with which the different standards were used. Some differences ; 

standards. Of more interest were the standards used when students were not 
/ given any guidelines as to the fcinds of things they should take into 
consideration. The lexical standard was very likely to be used by students 
• whether or not they were instructed to use it, indicating a basic concern with 

^ / the understanding of individual word meanings. However J students typically 
•\."' ! ,; did not evaluate what they were reading with respect to what they already 
; knew, even though they demonstrated use of this criterion when instructed to 
: ; ; use ; itf^: Approximately two thirds of the subjects receiving general 
= ; ^ infl^uattons~ne^ Students t#nd to v : 

accept the facts presented in texts at face value, a, tendency that 
automatically precludes any sort of critical reading. -Interestingly, this / 
; ; ? tendency may increase wi th development, as suggested by evidence that children / 

frequently challenge the '.truth df statements presented to them ih expository v ^ 
passage* even when they' are not; specifically instructed; to do so (e.g. , Baker* V 
Not# 1 ; Markriian, 1979)^ ; Whether or not this apparent decline in the use of 
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the, external consistency 3 

empirical question. ^If so, it may be an unfortunate by-product of 
lns trUQ ^ional ; approaches; s tr essing. that the answer : ia in toe book (Goodman V 
1976) . Ano ther dimension unlikely to be considered spontaneously is the 
internal consist Moreover, even when specific 

to evaluate for internal consistency, students* with lower verbal skills did 
not use ^e atandard 50$ or thg iubjWts inVtois 

group never applied the standard . A number of component skills are involved 
in the evaluation of internal consistency, including the abili ty to retrieve 
and integrate nonadjacent propositions and. draw logical inferences. These 
skills may well be limiting the lower verbal student 's use of the standard * " 

We have seen, then, that students spontaneously adopted a lexical 
^S^ 1 ?^?^ J?HS1 ^iM?H^ fi?^M^?rPfll or .in ternal consis tency s tandard - However , 
they did use another standard as frequently as the lexical : structural 
: cohesiyeness. This indicates a concern ^th the thematic relationships among 
the ideas- in ^the text and awareness of rhetorical conventions such as 
presenting topic sentences first. The standard is an impor tant one for 
students to apply, especially when studying , expository tex t, because %t 
requires the identification of the 2 main points of the passage. That 85? of 
the s tuden ts receiving general instructions adopted this standard is 
encouraging; however, it is possible that they used it only because they could 
not think of any other kinds of problems that might be present and they knew 
they had to identify something. A third standard likely to be adopted '■ 
spontaneously was informational completeness. Use of this standard reflects a 
concern wi th whether information is en ted in sufficient depth or ^ith 
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^sufficient clarity to ailow^one to understand i particul^ concept* ^ This 
standard is also impor tent whii# studying because it indicates to : the 

student whether there is a need to consult outside -sources for^ more 
forma tion, " " ..-"'-*•." •"=;.-.•' f. '•-••=•"*. ""- ••• r : " " •" 7"- ^r-rVv^-^ 

Rarely used were the prepositional cohesive^ syntactic standards. 

However, the ^passages did riot con tain any in ten tionally-in troduced prbblems- 
which Iwould elicit comments; reflecting the use of these standards and there 
were nit many Jnatur^ the 
study d^s nat prpvW answers as to the ex tent that these s tandards; 



are typical y used. We know that violations of propositional cohesiveness and 
syntactic constraints affect comprehension (e.g., Frazier & Rayner , 1982 • 
Hirst , & Brill\ 1980), but it may be that such discrepancies are processed 
?H^95^W?^liy And ,dp :nc^^^; require the deliberate evocation -of the relevant 
standards (cf. feaker . in press). - * - > • 

, ? n conclusion, the study has revealed both quantitative and qualitative 
differences in the standards used by college students to evaluate their 
understanding bf expository prose. It provides a rich source of hypotheses 
for fijJJ^ sensible and 

efficient criterW for evaluating their comprehension. For example, it 
illustrates the valAe of examinr&jg more syst>patically readers » skill at using 
all of the evaluation Ytandards under conditions designed to elicit their use. 
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.' 'V.-"* Examples of Comments Reflecting the Use r 

„ ~ \ of Different Standards of Evaluation 

Lexical -1 *v : ■ - tV : * * • - - - • - .' . " . -="'*"■-.-—- 

A. Pa ssage eon tex t i Some frogs defend themselves by feigning death . 

- " Comment: I don't think feigning is a word. * (specific instructions- 

lower verbal) . • •=.-: -. ~ • .;. . ; -;" . • ; ■•" - ; 4.';. ; V : '- - 1 = s .v !/• ' .V^L 

B. Passage context: Bats , the only group of flying mammals, are ; 
nocturnal insectivores. ■ \ !^ • L" 
Comment : I don f ;t know what insectivores are* (general instructions^ 

- . .... - — . : . -. , ................ ; .. .. ................ . . . . . ... . r . . . .. , 

External Consistency / ' • 

A # Passage context : Moves to get federal aid for the unemployed in the 
depressions of 1893-9^ » 1?1A. and 1921 „f ailed. * 



-\ Comment : I don't know my. history that well, but I don't think there 
were depressions all those; times, (specific instructions-rower verbal) 
B, Passage context: Franklin D. Roosevelt,, elected president in 1933, 
, immediately began to implement new relief programs. 
Comment : Elections are held in even-numbered years ; 1933 can f t be 
right, (general instructions-lower verbal) [This student, as well as 4 
othens, identified a problem embedded in the original t ex t and 

unnoticed \£y the experimenter - ] ■ - 

• .•• _ . ="•..*. • . 

■ ■ *■ ■ * j \ ■ ' ' 

Internal' Consistency v , . 

A- Passage context : Filled landfill sites are sometimes used for golf 

courses and parks- But building on such sites is hazardous because 5 of 

^ iiatural gas seepage from the underlying decaying garbage. 

Comment : Landfills are used for parks but they're too hazardous for 

' houses? That's stupid. (specific instructions, higher- verbal ) ; 

S« Passage context : Governors possess many formal powers to discharge the 

* duties of chief executive, chief legislator, and head of state. . . 



Evaluating exposi tory prose " 

V "Governors, frequently feel compelled to go. to great lengths to win 

legislators ov&r to their side. • / * ^ ~ ^ 7 .f " r " '" : 
- Comment: ;iV sayr g^ powers^ haVe rule over 

everyone. So why would they feel compelled ,to win legislators over to - 
v their : side? ; (specific ins tructxons-lower /verbal ) ^ .,. : *£-V;;y^7v. ^ , : 

Prbposi tional Cohesiveness " - - : . ... - V; - •'• ;.' ; ^ - ! - \ 

A. Passage eon text: Some [frogs] can blow up their IQnfts so that they are 
difficult to swallow. -•• - . : / :: V--/*:- /•-•■ 7':" 7 -'-7\^ : ;7 

Comment : They are difficult to swallow. What are? (specific 
instructions-higher verbal) V / 

B * Passage- con tex t r Sal es taxes" ar e lassessed^on al r those who buy tax abl e 
; goods f regardless of their residence. 

Comment : Their -residence. It almost sounds as though *the things 
- you're buying live somewhere else, (specific dnstructions-lower 
verbal ) "* \ - ' . "- 

' •' L° ■ -. * , }.' . ........ ..... . . .... . . t ^ . . . , , ........ 

Structural Cohesiveness 

A. Passage context : (The general topic of the passage is about the 

protective coloration of fish. The last two sentences appear here.) 
Some tropical fish are vividly colored » however, even in areas of dull 
and somber backgrounds. Al though conspicuous in these circumstances, 

they are protected either by alertness and agility or by their _ _ '.. 

poisonous flesh. . . . ... 

Comment 1 : Alertness and agility and poisonous flesh have nothing to do 
with color; the sentence is tacked onto the end. (general 
instructions-higher verbal) - 
B * Passage context: ^The passage is primarily concerned with the housing 
industry, but it begins with a comparison to the auto industry. > 
Comment : The title is Housing ajid Local Government* Then it goes into 
something about auto production which is totally irrelevant, (general 
instructions-lower verbal) 




A* • Passage con tex t : jm e general topic of the passage is pesticides and 

it includes a discussion of both her^b^ 
> Comment: I know insecticides 'arid herbicideg* ar^ hr>hh' f-.yp« - >_/ 
pesticides ■ anci. I know ho»w they differ ; but it!s not . specified in the 
: >;..;-)\ passage; the terms should be defined more clearly* ^specific - ' % 

' ; instructions-higher verbal ) r > y-/^ " V : V- •. .VI: •'=.•. 

®* Passage context . (The passage describes the special sense organs §f 
snakes, including hea The pits are supplied with a 

dense packing ofi free nerve endings from the fifth cranial nerve. ' ■ 
Comment : The passage doesn't explain what the fifth cranial nerve is. 
7 C^nefal in 



Syntax . " : '. •" ' : ■ ; ; . ; ' . ■ ' '■ iiV" 

A. Passage context : The. remaining 83 percent [of solid waste] is buried 
in landfill sites . with only about 10 percent of these being the 
so-called sanitary landfills. 

Comment : "With only about 11 is very awkward, (general instructions- 
lower verbal) " ^ ' .." ' ' ' ""' •• ' ' - 

B. Passage^oontext : Seen frW above against its normal background of 
^ water and stream bottom, the fish becomes almost invisible* 

Comment : "Seen from above 11 is hard to understand, there are three 
- prepositions together; I had to readmit three times, "(general 
instrubtions-iower verbal > /. • . ^ 



Parenthetical remarks indicate the type of instructions and the verbal 
ability level of the subject making the comment. 
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L ? . . : ' " - j ; Tab! © 2 * : ; / _ ' / 1 ^ . / j ' .. - 1" 

Mean Number of Problems Identified** " 

W". ;, ;v-.- . Problem Type ' •;.= ;; ; v "j 

Nonsense Prior Knowledge Internal 



7 





Words 


Violations Inconsistencies 


— -' i V'v ' - . : . Verbal * ; 








Instructions Ability , 








Specific Higher 


2.75 


1.17 


1.08 


Lower. 


1.88 


• ' " .75 


.19 ' . 


General Higher 


1.91 




.09 ■/.'.•:••''■:■ 


.'./. "••••. * Lower . .. 


' 1.63 


- _ " -26 | . 


.05 


Note: Maximum = 3v 


* 
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Table 3 



Mean Number of Tiroes Subjects Applied Each Standard . 



.Instructions 



Specific 



General 



Type of Standard 



; Hi.gh^r : Verbal -■- : ii^oworj ^V€^i>-«x- -^"vH± " ^i^w^^y^riMiX^ 
Lexical ^7 ^ -'i'' ^ " 3.19 3.46 

External Consistency 4.75 2.25 

Internal Consistency '• 2.33 * 



3.58 



.82 



Proposirtionai 
* Cohesiveness 

Structural 
Cohesiveness 

Informational 
Completeness 

Syntax 

Total 



.58 
1.00 

2.17 
.17 
15.17 



1.25 



.38 



2.00 



1 . 00 



,00 



10.07 



.45 
' . 82 
4.00 

1.82 

*36' 
11.73 



.63 
.26 



.53 
2.95 1 
' 1.00 

.74 

9.69 
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Appli ed a Par tioular Standard a t : l;#as t Onoe 



Type of Standard 
Lexical .-.. v " 



■ ■ ' • .- - .'• ; Instructions * ! '' • I V" 

Specific • ' ; ;;-; ; ; J ' : - General : v"" v , .•' 

Higher Verbal Lower Verbal j Higher Verbal Lower Verbal 



X. 00 



88 



.91 



89 



External Consistency 



In ter nal Consis tency 



.92 
-92 



,82 



.53 



Proposi tional 
Cohesivejness 



,42 



.24 



.36 



-36 



.26 



.26 



,16 



Structural 
Cohesiveness 



.58 



.82 



,82 



.89 



